…gentic configs)

Wire the SWE-bench Lite eval into the sweep matrix so it runs on agentic
coding configs, and route it through e2e-tests.

- utils/matrix_logic/generate_sweep_configs.py: add mark_eval_entries and
  mark_all_eval_entries. For agentic configs these mark exactly one eval
  entry per (model, runner, framework, precision) group at the highest
  concurrency, single-node only, so each unique agentic config gets one
  swebench eval run rather than one per concurrency point.
- utils/matrix_logic/test_generate_sweep_configs.py: add
  test_marks_agentic_entry_for_swebench and update TestMarkAllEvalEntries
  to cover the agentic marking behavior.
- .github/workflows/e2e-tests.yml: add the agentic-eval-config bucket, a
  test-sweep-agentic-evals job, and make collect-evals depend on it. The
  AGENTIC_EVAL filter (agentic + no prefill + run-eval) selects the eval
  entries; the throughput AGENTIC filter (agentic + not run-eval) excludes
  them so throughput and eval runs don't collide.
- benchmarks/single_node/agentic/kimik2.5_fp4_b300.sh: add the eval hook so
  the recipe triggers the agentic swebench eval.

…1.0) + bootstrap Modal creds from env

swebench 4.1.0 exposes --max_workers in both Docker and Modal modes; --parallelism
does not exist. Fix run_harness() to emit --max_workers in the Modal branch.

Add _ensure_modal_credentials() to benchmark_lib.sh: swebench's credential
check only looks for ~/.modal.toml, but CI supplies MODAL_TOKEN_ID/
MODAL_TOKEN_SECRET env vars (GitHub secret). The helper bootstraps the file
from the env vars when the file is absent, so the harness check passes. Called
in run_swebench_eval() right after _install_swebench_deps, scoring path only.

Update the Modal test name and assertions, the run_swebench_eval docstring,
and the EVALS.md knobs bullet to document the credential bootstrapping.

Apply the EVAL_ONLY=true if/else gating pattern (already present in
kimik2.5_fp4_b300.sh) to the remaining 24 single-node agentic recipes in
benchmarks/single_node/agentic/. In eval-only mode each recipe skips the
multi-turn agentic replay and calls maybe_run_eval "$PORT" against the live
server; run_eval auto-selects swebench for the agentic-coding scenario.
The deprecated/ subdirectory was not touched.

- Re-sync test-sweep-agentic-evals inputs with main's test-sweep-agentic:
  offloading -> kv-offloading + kv-offload-backend + total-cpu-dram-gb.
- Add EVAL_ONLY/maybe_run_eval tail gating to the agentic recipes AgentX
  v1.0 added (dsv4_fp4_b200_sglang, dsv4_fp4_b300_sglang, minimaxm3_fp8_h100/
  h200/mi300x/mi325x) so eval-only runs skip the replay like the others.
- test_run_eval_dispatch: set KV_OFFLOADING=none so the new source-time
  agentic guard in benchmark_lib.sh is satisfied (dispatch logic unaffected).

…dling

Add EVAL_LIMIT env var to run_lm_eval() so --limit N is appended to the
lm_eval invocation when set, enabling small smoke runs (e.g. 10 instances)
without touching the full ~300-instance swebench suite. Wire the knob through
benchmark-tmpl.yml (new eval-limit input + EVAL_LIMIT env) and e2e-tests.yml
(both workflow_dispatch and workflow_call inputs; passed through to
test-sweep-evals and test-sweep-agentic-evals with: blocks). Document the
variable in utils/evals/EVALS.md.

Harden _ensure_modal_credentials against b300 slurm/pyxis containers where
--export=ALL propagates the HOST's HOME into the container; if HOME is unset,
mkdir -p fails, or the directory isn't writable, remap HOME to
/tmp/inferencex-modal-home before writing ~/.modal.toml. Remap is scoped to
the write path (SWEBENCH_USE_MODAL=true, file absent, tokens present).

Tests: functional shim tests for --limit presence/absence; HOME-remap tests
covering writable home (no remap), read-only parent (remap + 600 perms), and
non-writable existing dir (remap); and a no-op test when SWEBENCH_USE_MODAL=false.

…mpty SWEBENCH_NAMESPACE arg

- Add `include_agentic: bool = False` to `mark_eval_entries`; wrap the
  `ag_sn_groups` agentic-marking block in `if include_agentic:` so that
  default sweeps no longer set `run-eval: true` on any agentic entry.
  The e2e-tests.yml AGENTIC filter (`not x.get('run-eval', False)`) then
  routes all agentic entries to the throughput job, restoring main parity.
- Pass `include_agentic=args.evals_only or args.all_evals` in `main()` so
  --evals-only and --all-evals continue to mark and select agentic entries.
- Replace `${SWEBENCH_NAMESPACE+--namespace "$SWEBENCH_NAMESPACE"}` with an
  `ns_args` array in `run_swebench_eval`; when `SWEBENCH_NAMESPACE=""` the
  old form word-split to a bare `--namespace` (argparse error); the array
  form safely expands `--namespace ""` or nothing when unset.
- Tests: `test_marks_agentic_entry_for_swebench` updated to pass
  `include_agentic=True`; new `test_default_mode_does_not_mark_agentic`
  asserts zero agentic entries marked in default mode; new ns_args unit
  tests cover unset/empty/value cases plus a static assertion that the old
  pattern is gone from benchmark_lib.sh.

… KeyErrors on unregistered task-name paths)

The pinned lm-eval (0.4.9.2, ref b315ef3) crashes with
KeyError: '<task_name>' in pretty_print_task (tasks/__init__.py:681) when
--tasks is given a file path to an external YAML whose task: name is not in
lm-eval's bundled registry.  gsm8k/gpqa_diamond are immune because those
names exist in the bundled registry; swebench_lite is not.

Fix: in run_lm_eval(), add optional EVAL_INCLUDE_PATH support — when set,
injects --include_path "$EVAL_INCLUDE_PATH" just before --tasks; inert when
unset (gsm8k/gpqa production invocations are byte-identical).

In run_swebench_eval(), switch the generation call from
  EVAL_TASKS_DIR="$yaml_path"     (path form → KeyError)
to
  EVAL_TASKS_DIR="$task_name"     (name form)
  EVAL_INCLUDE_PATH="$(dirname "$yaml_path")"   (registers the dir)
with save/restore of both vars so EVAL_INCLUDE_PATH does not leak to
subsequent lm-eval invocations.  The dataset_path-from-YAML derivation
(awk over yaml_path) is unchanged — generation and scoring remain in lockstep.

Tests: two shim-based dynamic tests (EVAL_INCLUDE_PATH set/unset → flag
present/absent in argv; --tasks carries name vs. yaml path) and one static
assertion that run_swebench_eval source contains EVAL_INCLUDE_PATH wiring.

…iling newline fail validation)

Live probe proved it: MODAL_TOKEN_SECRET secret has a trailing whitespace char;
raw auth fails ('Token validation failed'), whitespace-stripped auth succeeds.
Strip whitespace/quotes and re-export in _ensure_modal_credentials so both the
modal client (env) and the bootstrapped ~/.modal.toml are clean.

…lure

- run_swebench_eval: wrap scoring in timeout ${SWEBENCH_SCORE_TIMEOUT:-7200}s.
  The overnight 300-instance run stalled ~7h in Modal image builds and held the
  b300 allocation until the slurm wall; a stalled backend now fails fast.
- maybe_run_eval: always stage eval artifacts (append_lm_eval_summary) even when
  the eval fails, then propagate the rc — samples/predictions survive for
  diagnosis instead of dying in the job sandbox.

…ation input)

Agent harnesses (SWE-agent / mini-swe-agent) emit standard predictions.jsonl
directly; this bypasses lm-eval samples parsing and feeds the existing Modal
scoring + results pipeline unchanged. Groundwork for agentic swebench.

…sandboxes

SWEBENCH_GEN_MODE=agentic runs a real agent loop per instance instead of the
single-shot prompt: mini-swe-agent (2.4.5) drives the local OpenAI-compatible
endpoint; each instance's shell executes in a Modal sandbox (swe-rex[modal],
official swebench per-instance images -- no docker needed on the GPU node).
preds.json feeds the existing Modal scoring via --predictions-file (which now
also accepts the dict-keyed preds.json format directly).

- benchmark_lib.sh: _run_swebench_agentic_generation (config overlay, slice via
  EVAL_LIMIT, workers/step/timeout knobs), _install_swebench_agent_deps
  (mini-swe-agent==2.4.5 + swe-rex[modal]==1.4.0), gen-mode branch in
  run_swebench_eval feeding scoring via score_input array.
- swebench_score.py: --predictions-file accepts dict preds.json or JSONL.
- workflows: swebench-gen-mode input threaded e2e-tests -> benchmark-tmpl env.
- tests: shim-driven agentic-generation test + predictions-file format tests.

Single-shot remains the default; agentic is the real SWE-bench setting.

Fresh installs print a multi-line version banner on import; take only the last
stdout line and validate it is a file. Shim test now emulates the banner.

mini's default startup_timeout=60s is consumed by the cold GB-scale swebench
image pull alone ('Runtime did not start within 0s'). Default 900s via
SWEBENCH_AGENT_STARTUP_TIMEOUT; command timeout 300s (mini default 60s is too
tight for running repo test suites) via SWEBENCH_AGENT_CMD_TIMEOUT.

Trajectories are the primary forensic artifact for agent tuning; they
previously died with the job's temp dir. Copy *.traj* flat into the eval
output (append_lm_eval_summary flattens *.json* into the workspace root),
upload via new globs, and clean up post-upload.

Findings from 10-trajectory deep-dive (first-10 Lite, DSv4):
- 3/5 unresolved agents submitted without ever running the failing test
- 1 agent had the CORRECT fix on disk at step 31, burned 44 steps fighting an
  unfixable sandbox C-extension build, and hit the step cap without submitting
- CoT leaks into visible content (deepseek_v4 reasoning parser init failure,
  recipe-side follow-up) -- 'execute over prose' guidance mitigates

Replace the static config heredoc with a runtime merger that appends targeted
guidance to mini's instance_template: verify-before-submit, build-failure
escape hatch, submission discipline, step-budget framing. Single merged config
replaces the dual -c chain.

Every agent sandbox was billing a full hour for ~7-minute instances
(observed: batches dying at 59m59s on the Modal dashboard). Three leaks:

- mini-swe-agent 2.4.5 process_instance() never calls env.stop(), even on
  success, so every sandbox lives until runtime_timeout (3600s default).
- swe-rex 1.4.0 ModalDeployment.stop() has its poll check inverted: it
  terminates only sandboxes that already exited and skips running ones.
- ModalDeployment.start() leaks the sandbox when the runtime never comes
  alive (the startup-timeout failure mode).

Fix: _patch_swebench_agent_cleanup() patches the installed files at dep
install (idempotent, anchor-checked against the pinned versions) so
sandboxes terminate the moment their instance finishes; a post-generation
workspace sweep reaps anything that slips through (crashed workers, outer
timeout kills; SWEBENCH_SANDBOX_SWEEP=0 disables for tests); and the
merged config now sets runtime_timeout explicitly
(SWEBENCH_AGENT_RUNTIME_TIMEOUT, default 3600) as a pure backstop.

No agent-visible behavior change: cleanup happens after instance
completion, so resolved-rate comparisons across runs stay clean.

… budget exhaustion

Run-1/3 findings (50 instances, tuned template):

- Metric bug: the harness report's total_instances is the full dataset size
  (300) even with EVAL_LIMIT=50, so a 32/50 (64%) run was published as
  0.107 and nearly tripped the 0.10 threshold gate. parse_resolved now
  prefers submitted_instances over total_instances (identical for
  full-split runs).

- 6/50 instances hit LimitsExceeded after 75 steps and submitted NOTHING,
  despite forensics showing fixes can be complete mid-run. patched
  process_instance now falls back to submitting `git diff` of the working
  tree when an instance ends abnormally with a live sandbox (requires rc 0
  and a `diff --git` prefix so an error string can never become a patch).
  Empty submissions score zero, so the fallback is strictly >=.

- Stage the swebench harness report as swebench_report_<task>.json and
  upload it; it names resolved/unresolved per instance and was previously
  left behind on the node.

Run-2/3 verified the sandbox-cleanup patches (applied on the node, sweep
found 0 lingering sandboxes) but 0 fallback submissions fired while 6
instances still ended LimitsExceeded with empty patches. Root cause: mini's
agent run loop absorbs InterruptAgentFlow (Submitted, LimitsExceeded, ...)
and RETURNS normally with an empty submission -- LimitsExceeded never
reaches process_instance's except branch, which is where the fallback hook
lived (their trajectories carry no traceback/exception_str keys,
confirming the normal-return path).

Move the primary hook to just after agent.run(): any empty submission with
a live sandbox now submits `git diff` of the tree (same rc-0 +
"diff --git"-prefix guards). The except-path hook stays for real
exceptions.

Two full-300 Modal scorings measured ~$80 each in eval sandboxes alone
(vs $0.99-5.91 for image builds -- caching was never the cost driver).
Root cause: swebench's run_evaluation_modal.py hardcodes cpu=4 per
sandbox; Modal bills reserved cores and the test runs are predominantly
single-threaded pytest.

Patch the installed file at dep install (idempotent, anchor-checked,
numeric-validated) to SWEBENCH_EVAL_SANDBOX_CPU (default 2). Per-instance
tests run somewhat slower on fewer cores; scoring parallelism absorbs it.

…l default (75)

…ng utilization)

run_instance_modal never finalizes its ModalSandboxRuntime -- the __exit__
that terminates the sandbox exists but nothing calls it -- so every eval
sandbox idle-bills after its tests finish until the 30-min sandbox timeout
or ephemeral-app teardown. Invisible on the 50-slice (all-fast tests, the
app ends in ~2 min and reaps everything); on full-300 the slow tail keeps
the app alive ~40 min and all 300 sandboxes bill ~30 min for ~3 min of
work: measured 152 sandbox-hours ($41.57 at cpu=2) where real test time is
~15-20 sandbox-hours.

Patch (same install-time mechanism, idempotent, anchor-checked): a
finally: on run_instance_modal's main try/except chain terminates the
sandbox on every exit path. Expected full-300 scoring: ~$41 -> ~$5-8.
_patch_swebench_scoring_cpu renamed _patch_swebench_scoring (cpu +
lifecycle hunks).

…andboxes

Modal's default sandbox reservation is fractional-core, and the agents run
real test suites inside these sandboxes (verify-before-submit), where a
starved CPU can eat the 300s command timeout and waste agent steps. Optional
knob threads through mini's modal_sandbox_kwargs -> swe-rex ->
modal.Sandbox.create; unset preserves the Modal default (current behavior).
Added for the cost/time pareto sweep.

…imeout

Post-lifecycle-patch, real test runs bill ~33s each and the scoring wall
(~29 min on full-300) is set almost entirely by the 7 persistently-erroring
instances running to the harness's 1800s default. Optional pass-through to
run_evaluation --timeout; unset preserves the harness default.

… rename, app-scoped sweep

Pareto-sweep conclusions (11 runs):

- SWEBENCH_AGENT_WORKERS defaults to the config's CONC (else 64): the eval
  drives the server at the concurrency its config was tuned for. At conc144
  this cut full-run generation 90m -> 51m at identical score (five full
  runs: 160-163/300); the old w144 hang risk is contained by the completion
  watchdog (three clean w144 runs since).
- SWEBENCH_EVAL_TIMEOUT defaults to 900s: real test runs bill ~33s; only
  the persistently-erroring instances touch the ceiling and they gate the
  scoring tail.
- swe-rex's hardcoded Modal app name is patched to SWEBENCH_MODAL_APP_NAME
  (default infx-evals-swe) so the dashboard shows ours, not the library's.
- The post-generation sweep is now scoped to that app via
  Sandbox.list(app_id=...) -- it can no longer touch other apps' sandboxes
  in the shared workspace (narrows the concurrent-tenant hazard to
  same-app legs only).

Agent-sandbox CPU knob stays unset by default: the sweep measured ~zero
command timeouts at Modal's fractional-core default (1 in 300 on the full
set) -- the agent loop is inference-bound and boosting is pure cost.

The SWE-bench eval patches three pinned eval-tooling packages at install
time (mini-swe-agent, swe-rex, swebench harness) -- sandbox-lifecycle and
cost fixes measured at ~17x Modal spend reduction. No inference engine or
serving stack is touched; the waiver is filed proactively because the
mechanical shape (heredoc patches in benchmark_lib.sh) matches Check 9's
inline-patch pattern.

…gine/serving patches (vendor patchwork), not our own eval-harness tooling; vLLM runs as shipped throughout

…fresh EVALS.md

Merge-readiness sweep findings:

- BLOCKER: dsv4_fp4_mi355x_vllm.sh arrived via a main-merge (#2109) after the
  eval-gating rollout, so it was the only 1 of 23 single-node agentic recipes
  without the EVAL_ONLY/maybe_run_eval block. A live config targets it
  (configs/amd-master.yaml: dsv4-fp4-mi355x-vllm-agentic), which the generator
  marks for eval -- under EVAL_ONLY=true the recipe would fall through to the
  throughput replay and produce no score (failing the eval-scores gate).
  Appended the standard gating block (matches its sglang sibling).
- swebench install pinned to ==4.1.0: the harness CLI flags and the
  _patch_swebench_scoring anchors are verified against 4.1.0; an unpinned
  upgrade could drift either.
- EVALS.md refreshed to the shipped behavior: agentic-only default, 0.50 gate,
  50-slice/full run sizing, and the current knob set/defaults.

…guard test)

Second readiness sweep returned GO (no blockers; mi355x gate fix held).
Clearing the actionable nits:

- EVALS.md: --all-evals no longer says agentic configs are excluded (they're
  included and run swebench under evals-only/all-evals; excluded only from the
  default non-eval sweep).
- generate_sweep_configs.py: three "(single-shot)" comments corrected to
  agentic (generation is agentic-only).
- benchmark-tmpl.yml: eval-limit input description corrected (empty = 50-slice
  swebench default, not "full set").
- tests: cover the EVAL_LIMIT positive-integer rejection guard (-5/abc/3.5 ->
  fail fast) and the full/0 whole-split sentinels. 53 eval tests pass.

modal left unpinned deliberately: unlike the three source-patched packages, it
is a client to a live hosted service where an exact pin invites client/server
skew.